Timeline resource streaming#1929
Conversation
5f1da0c to
24611dc
Compare
|
I've been testing this with @AaronPlave and we noticed something - the UI streaming works fine for normal simulation runs, but not for scheduling runs which run simulation at the end. For a (relatively) big plan, we're able to click simulate and see all the profiles slowly load in for the normal case. When I click "Schedule" with some goals in place, I see a couple of profiles load completely first, the sim hangs at "0%" for awhile, then the full remaining profiles all load at once. @Mythicaeda does this ring any bells for you? Do simulations triggered by scheduling stream their profiles to the backend the same way that normal runs do? Or does this seem like a likely UI bug to you? |
dandelany
left a comment
There was a problem hiding this comment.
Testing note - the timeline loading indicator at the top of the timeline is shown when simulation is running, but not during Scheduling. Per discussion we think it's best to also show it during Scheduling, since that is what is ultimately blocking the Timeline from showing up.
@AaronPlave will get to this tomorrow if he has time - otherwise we consider this a nonblocker for later
…tch instead to reduce complexity and brittleness.
…riptions Replace batch profile fetching with per-(datasetId, name) subs that pull only segments added since the last seen start_offset. Sim profiles and external dataset profiles share one shape (subscribe → render → unsubscribe) on top of a ref-counted resource-status registry that a new TimelineStatusIndicator reads.
…at external event states are globally tracked as one group, not individual derivation group states
5aa2846 to
cbf1b57
Compare
…indicator status computation
|
Done |
@dandelany This is not a UI bug. Scheduling does not stream to the database. Explanation as to why: (Hopefully it's intuitively understandable why we don't stream the scheduler's "working simulations" to the database, but just in case I'll give a couple reasons:
|



Summary
This PR implements streaming of simulation resources and external datasets (profiles) from the database as they are available, closes #1650 . External datasets are now fetched on an as-needed basis instead of fetching all external datasets that could possibly be associated with the plan every time
planDatasetsubscription changed, closes #352.Sim profiles stream segments via a new
createProfileSubscriptionfactory keyed on(datasetId, name). EachsimulationDatasetupdate triggersGET_PROFILE_SINCEfor only the segments past the last seenstart_offset.External dataset profiles stream the same way via a new
createExternalResourceSubscriptionfactory. Metadata source isplanDatasets(live subscription, already in use). When a profile'sdurationadvances, the factory pulls only the new segments viaGET_EXTERNAL_PROFILE_SEGMENTS_SINCE.Removes the batch external profiles fetch.
GET_PROFILES_EXTERNAL,effects.getResourcesExternal, theexternalResources/fetchingResourcesExternalwritable stores, and the +page.svelte reactive statement that re-fetched all external profiles on everyplanDatasetspush are all gone.Fixes external profile start time anchor modify external profile start time to use plan start time instead of sim start time so that external profiles draw correctly in time when plan is using subset sim.
New
TimelineStatusIndicatornext to the Timeline header. Aggregates loading/error across profile subscriptions, span fetches, constraint runs, external-event raw subscriptions, and active simulation status. Tooltip shows what failed (sim "Profile X" vs external "External profile X").stores/timelineResourceStatus.tsloading/error registry for any resource on the timeline.allResourceTypesnow derives fromplanDatasetsmetadata directly (filtered to plan-level + current-sim rows) instead of the now removed batch fetch.External event row loading statesloading/error states on rows containing external events. Note that this is derived from the global external sources store since there are no individual fetches or subscriptions for singular derivation groups so if one is loading/failed then all will share that state.Tests
stores/profile.test.ts,stores/externalResource.test.ts,utilities/profile.test.ts. Cover settling→found, race-defer, settled-missing, sim-tied vs plan-level row preference, duration-advance refetch, profile-row switch reset, refetch error surface, dispose-during-in-flight.utilities/resources.test.tswith sort-invariant regression pins onsampleProfiles.simulation.test.tsStreaming pipeline smoke test — asserts the indicator settles and timeline canvases render non-transparent content across two re-sims. Shared canvas-pixel helper ine2e-tests/utilities/canvas.tsis now used byplan-external-source.test.tstoo.Test plan